Towards Cloud Application Description Templates Supporting Quality of Service

Typical scientific, industrial and public sector applications require resource scalability and efficient resource utilization in order to serve a variable number of customers. Cloud computing provides an ideal solution to support such applications. However, the dynamic and intelligent utilization of cloud infrastructure resources from the perspective of cloud applications is not trivial. Although there have been several efforts to support the intelligent and coordinated deployment, and to a smaller extent also the run-time orchestration of cloud applications, no comprehensive solution has emerged until now that successfully leverages large scale near operational levels and ease of use. COLA is a European research project to provide a reference implementation of a generic and pluggable framework that supports the optimal and secure deployment and run-time orchestration of cloud applications. Such applications can then be embedded into workflows or science gateway frameworks to support complex application scenarios from user-friendly interfaces. A specific aspect of the cloud orchestration framework developed by COLA is the ability to describe complex application architectures incorporating several services. Besides the description of service components, the framework will also support the definition of various Quality of Service (QoS) parameters related to performance, economic viability and security. This paper concentrates on this latter aspect analysing how such application description templates can be developed based on existing standards and technologies

High-level Description of Cloud Applications using TOSCA

Public sector organizations and SMEs are increasingly considering using cloud services in their everyday activities. At the one hand on-demand access to cloud services in a flexible and elastic way could result in significant cost savings due to more efficient and convenient utilization. Further it can also replace large investment costs with long-term operational costs. On the other hand, the take up of cloud computing by the public sector and Small- and Medium-size Enterprises (SME) is still relatively low due to limited application-level flexibility and shortages in cloud specific skills. To meet these requirements a generic framework is needed to support public sector organizations and SMEs to run large variety of applications in the cloud in a cost effective, flexible and seamless way. To address these challenges the European funded COLA (Cloud Orchestration at the Level of Application) [1] project is designing and developing a modular architecture called MiCADO (Microservices-based Cloud Application-level Dynamic Orchestrator) [2]. It provides optimized deployment and run-time orchestration for cloud applications. MiCADO can manage applications considering their specific deployment, execution, scalability and security requirements. To further address this challenge COLA uses TOSCA (Topology and Orchestration Specification for Cloud Applications [3] to describe applications to be executed in the cloud. Application developers can create so called Application Description Templates (ADT) to specify and submit their applications to the cloud through MiCADO. ADTs define two key properties of applications: topologies and policies. There are two approaches to define ADTs: using either command-line interfaces or graphical user interfaces. Command-line interface requires deep knowledge of the TOSCA specification and good YAML knowledge. Since application developers in the public sector organizations and at SMEs may not have this knowledge COLA’s priority is providing a GUI–based environment to enable application developers to describe their applications. The project investigated several GUI-based TOSCA development environments such as, OpenTOSCA Winery [4] and Alien 4 Cloud [5]. Winery generates XML-based specification of application topologies. The current Winery version automatically translates the XML-based TOSCA specifications into YAML to make them compatible with the latest TOSCA specification. Since each translation has its own limitations, some TOSCA features that are required in COLA, are lost in translation. The other limitation of Winery is that it does not support the definition of TOSCA policy specifications. Fig. 1 presents a simple topology template developed in Winery. Although Alien 4 Cloud supports the definition of cloud applications through a GUI environment, the generated description is not fully TOSCA compliant and cannot be parsed with most widely used TOSCA parsers. Considering the above listed limitations, COLA is developing a GUI-based environment to support application specification in TOSCA YAML v1.0. The extra feature of this environment will be a wide range support for policy specification, for example enabling development of deployment, execution, scalability and security policies

Enabling modular design of an application-level auto-scaling and orchestration framework using tosca-based application description templates

This paper presents a novel approach to writing TOSCA templates for application reusability and portability in a modular auto-scaling and orchestration framework (MiCADO). The approach defines cloud resources as well as application containers in a flexible and generic way, and allows for those definitions to be extended with specific properties related to a desired container orchestrator chosen at deployment time. The approach is demonstrated in a proof-of-concept where only a minor change was required to a previously used application template in order to achieve the successful deployment and lifecycle management of the popular web authoring tool Wordpress on a new realization of the MiCADO framework featuring a different container orchestrator

Describing and Processing Topology and Quality of Service Parameters of Applications in the Cloud

Typical cloud applications require high-level policy driven orchestration to achieve efficient resource utilisation and robust security to support different types of users and user scenarios. However, the efficient and secure utilisation of cloud resources to run applications is not trivial. Although there have been several efforts to support the coordinated deployment, and to a smaller extent the run-time orchestration of applications in the Cloud, no comprehensive solution has emerged until now that successfully leverages applications in an efficient, secure and seamless way. One of the major challenges is how to specify and manage Quality of Service (QoS) properties governing cloud applications. The solution to address these challenges could be a generic and pluggable framework that supports the optimal and secure deployment and run-time orchestration of applications in the Cloud. A specific aspect of such a cloud orchestration framework is the need to describe complex applications incorporating several services. These application descriptions must specify both the structure of the application and its QoS parameters, such as desired performance, economic viability and security. This paper proposes a cloud technology agnostic approach to application descriptions based on existing standards and describes how these application descriptions can be processed to manage applications in the Cloud

Semantic Data Pre-Processing for Machine Learning Based Bankruptcy Prediction Computational Model

This paper studies a Bankruptcy Prediction Computational Model (BPCM model) – a comprehensive methodology of evaluating companies’ bankruptcy level, which combines storing, structuring and pre-processing of raw financial data using semantic methods with machine learning analysis techniques. Raw financial data are interconnected, diverse, often potentially inconsistent, and open to duplication. The main goal of our research is to develop data pre-processing techniques, where ontologies play a central role. We show how ontologies are used to extract and integrate information from different sources, prepare data for further processing, and enable communication in natural language. Using ontology, we give meaning to the disparate and raw business data, build logical relationships between data in various formats and sources and establish relevant context. Our Ontology of Bankruptcy Prediction (OBP Ontology) which provides a conceptual framework for companies’ financial analysis, is built in the widely established Prote ́ge ́ environment. An OBP Ontology can be effectively described with a graph database. Graph database expands the capabilities of traditional databases tackling the interconnected nature of economic data and providing graph-based structures to store information allowing the effective selection of the most relevant input features for the machine learning algorithm. To create and manage the BPCM Graph Database (Graph DB), we use the Neo4j environment and Neo4j query language, Cypher, to perform feature selection of the structured data. Selected key features are used for the Machine Learning Engine – supervised MLP Neural Network with Sigmoid activation function. The programming of this component is performed in Python. We illustrate the approach and advantages of semantic data pre-processing applying it to a representative use case

Flexible Deployment of Social Media Analysis Tools, Flexible, Policy-Oriented and Multi-Cloud deployment of Social Media Analysis Tools in the COLA Project

The relationship between companies and customers and among public authorities and citizens has changed dramatically with the widespread utilisation of the Internet and Social Networks. To help governments to keep abreast of these changes, Inycom has developed Eccobuzz and Magician, a set of web applications for Social Media data mining. The unpredictable load of these applications requires flexible user-defined policies and automated scalability during deployment and execution time. Even more importantly, privacy norms require that data is restricted to certain physical locations. This paper explains how such applications are described with Application Description Templates (ADTs). ADTs define complex topology descriptions and various deployment, scalability and security policies, and how these templates are used by a submitter that translates this generic information into executable format for submission to the reference framework of the COLA European projec

Live Demonstration of the PITHIA e-Science Centre

PITHIA-NRF (Plasmasphere Ionosphere Thermosphere Integrated Research Environment and Access services: a Network of Research Facilities) is a four-year project funded by the European Commission’s H2020 programme to integrate data, models and physical observing facilities for further advancing European research capacity in this area. A central point of PITHIA-NRF is the PITHIA e-Science Centre (PeSC), a science gateway that provides access to distributed data sources and prediction models to support scientific discovery. As the project reached its half-way point in March 2023, the first official prototype of the e-Science Centre was released. This live demonstration will provide an overview of the current status and capabilities of the PeSC, highlighting the underlying ontology and metadata structure, the registration process for models and datasets, the ontology-based search functionalities and the interaction methods for executing models and processing data. One of the main objectives of the PeSC is to enable scientists to register their Data Collections, that can be both raw or higher-level datasets and prediction models, using a standard metadata format and a domain ontology. For these purposes, PITHIA builds on the results of the ESPAS FP7 project by adopting and modifying its ontology and metadata specification. The project utilises the ISO 19156 standard on Observations and Measurements (O&M) to describe Data Collections in an XML format that is widely used within the research community. Following the standard, Data Collections are referring to other XML documents, such as Computations that a model used to derive the results, Acquisitions describing how the data was collected, Instruments that were used during the data collection process, or Projects that were responsible for the data/model. Within the XML documents, specific keywords of the Space Physics ontology can be used to describe the various elements. For example, Observed Property can be Field, Particle, Wave, or Mixed, at the top level. When preparing the XML metadata file, only these values are accepted for validation. Once described in XML format, Data Collections can be published in the PeSC and searched using the ontology-based search engine. Besides large and typically changing/growing Data Collections, PeSC also supports the registration of Catalogues. These are smaller sets of data, originating from a Data Collection and related to specific events, e.g. volcano eruptions. Catalogue Data Subsets can be assigned DOIs to be referenced in publications and provide a permanent set of data for reproducibility. Additionally, to publication and search, the PeSC also provides several mechanisms for interacting with Data Collections, e.g. executing a model or downloading subsets of the data. In the current version two of the four planned interaction methods are implemented: accessing the Data Collection by a direct link and interacting with it via an API and an automatically generated GUI. Data Collections can either be hosted by the local provider or can be deployed on EGI cloud computing resources. The development of the PeSC is still work in progress. Authentication and authorisation are currently being implemented using EGI Checkin and the PERUN Attribute Management System. Further interaction mechanisms enabling local execution and dynamic deployment in the cloud will also be added in the near future. The main screen of the PeSC is illustrated on Figure 1. The source code is open and available in GitHub

Towards a Cloud Native Big Data Platform using MiCADO

In the big data era, creating self-managing scalable platforms for running big data applications is a fundamental task. Such self-managing and self-healing platforms involve a proper reaction to hardware (e.g., cluster nodes) and software (e.g., big data tools) failures, besides a dynamic resizing of the allocated resources based on overload and underload situations and scaling policies. The distributed and stateful nature of big data platforms (e.g., Hadoop-based cluster) makes the management of these platforms a challenging task. This paper aims to design and implement a scalable cloud native Hadoop-based big data platform using MiCADO, an open-source, and a highly customisable multi-cloud orchestration and auto-scaling framework for Docker containers, orchestrated by Kubernetes. The proposed MiCADO-based big data platform automates the deployment and enables an automatic horizontal scaling (in and out) of the underlying cloud infrastructure. The empirical evaluation of the MiCADO-based big data platform demonstrates how easy, efficient, and fast it is to deploy and undeploy Hadoop clusters of different sizes. Additionally, it shows how the platform can automatically be scaled based on user-defined policies (such as CPU-based scaling)